Apparatus and method for determining a location in a target image

ABSTRACT

An apparatus and a computer-implemented method are provided for determining a location in a target image (T) of a site on a surface of a physical object using two or more reference images (I 1 , I 2 ) of said physical object that have been obtained with a reference imaging device. Each of said two or more reference images includes said site on the surface of the physical object and was obtained with the reference imaging device having a different position and/or orientation relative to said physical object. The target image is obtained by a target imaging device and includes the site on the surface of the physical object. For each reference image, a set of feature mappings from the reference image to the target image is used to determine the epipolar geometry between the reference image and the target image, and a projection of the site from the reference image onto the target image is calculated from said epipolar geometry. The location in the target image of the site on the surface of the physical object is determined from the calculated epipolar projections for the two or more reference images.

FIELD OF THE INVENTION

The present invention relates to an apparatus and method for determininga location in a target image of a site on a surface of a physicalobject.

BACKGROUND OF THE INVENTION

Endoscopy is a minimally invasive procedure for the real-timeacquisition of video images of the interior surfaces of an organ with aflexible or rigid endoscope. Endoscopy is often used to allow a biopsyto be taken. Significant problems for many endoscopies are biopsy sitedetection, path planning to reach the site, and re-localisation(re-identification) of the site during the same or a future examination.For example, many endoscopic procedures (such as oesophageal endoscopyfor Barrett's Oesophagus) require the endoscopist to return to alocation previously identified to take a measurement or extract a smallsample of tissue (biopsy) for analysis of cellular structure and/or todetect the presence of pathology, in particular cancer and pre-cancerousconditions. Re-localisation may be required during the procedure as theendoscope may move, the patient may cough etc, or it may be required ata later date in order to assess disease progression.

One particular method is known as optical biopsy. This method is basedon the properties of light to make a diagnosis in vivo and in situduring endoscopy, while the diagnosis is traditionally done byhistological or cytological analysis. It has been shown that opticalbiopsies contribute to a better detection of malignancy which isinvisible during endoscopy and improve the accuracy of the diagnosis.However, it is difficult in practice to have a good match between atissue sample extracted for histology and the extent scanned duringoptical biopsy. As a result, many extracted samples may presentirrelevant information, which brings great difficulties to theclinicians to diagnose, screen, stage and treat diseases.

In bronchoscopy, for example, an attempt has been made to solve theseproblems with a guidance system based on tracking the bronchoscopeduring the examination in relation to a pre-operative CT image (seeHelferty J. P. et al., “Computer-based system for the virtual-endoscopicguidance of bronchoscopy”, October-November 2007, Computer Vision andImage Understanding, Vol. 108, Issues 1-2, pp. 171-187; and Mori K. etal., “Bronchoscope tracking without fiducial markers using ultra-tinyelectromagnetic tracking system and its evaluation in differentenvironments”, October 2007, Medical Image Computing andComputer-Assisted Intervention (MICCAI'07), Vol. 4792 of Lecture Notesin Computer Science, pp. 664-651, Springer-Verlag). Such systems havehelped to localise biopsy sites with a precision of 1.58 mm (Helferty J.P. et al.). Nevertheless, some examinations like gastroscopy are basedonly on video images. Therefore, the endoscopist may want to useminiprobes that are inserted into the working channel of the endoscopeand that return additional microscopic information (in vivo histology)or any other signal from the tissue in order to detect and localisebiopsy sites. When the endoscopist detects a suspicious region, it isscanned with the miniprobe in contact with the tissue surface in orderto confirm the diagnosis. If a tissue sample needs to be extracted fromthis region, the miniprobe is replaced with forceps.

A significant problem for applications in endoscopy procedures based onvideo images only and for gastroscopy in particular is that biopsies areperformed by interactive guidance of the forceps by the endoscopist.Therefore, once a biopsy site has been detected with a miniprobe, forexample, at the tissue surface, it needs to be re-localised precisely inthe next video images in order to go back to the same position with theforceps. The problem of object localisation in video images has recentlybeen addressed in applications for minimally invasive surgery (seeSpeidel S. et al., “Tracking of Instruments in Minimally InvasiveSurgery for Surgical Skill Analysis”, MIAR 2006, Vol. 4091 of LectureNotes in Computer Science, pp. 148-155, Springer-Verlag; and Voros S. etal. “Automatic Localization of Laparoscopic Instruments for the VisualServoing of an Endoscopic Camera Holder”, 2006, Medical Image Computingand Computer-Assisted Intervention (MICCAI'06), Vol. 4190 of LectureNotes in Computer Science, pp. 535-542, Springer-Verlag). However, thesedocuments focus only on tracking the instrument tip and its trajectoryfor robotized-assisted surgeries.

The above problems are also relevant to other medical applications wherea point site needs to be re-localised, for example for accurate therapydelivery. In addition, similar problems also apply to non-medicalapplications such as industrial inspection; remote sensing in hostileenvironments or very remote sites, for example, underground orunderwater, or in space exploration; remote manipulation and repair; andtelerobotics.

SUMMARY OF THE INVENTION

One embodiment of the invention provides a computer-implemented methodfor determining a location in a target image (T) of a site on a surfaceof a physical object. The method includes providing two or morereference images (I₁, I₂) of said physical object that have beenobtained with a reference imaging device. Each of the two or morereference images includes said site on the surface of the physicalobject and was obtained with the reference imaging device having adifferent position and/or orientation relative to said physical object(compared with its position for the other reference images). The methodfurther includes receiving a target image obtained by a target imagingdevice (which may be the same device as the reference imaging device ora different device). The target image includes the site on the surfaceof the physical object.

For each reference image, the method then includes using a set offeature mappings from the reference image to the target image todetermine the epipolar geometry between the reference image and thetarget image, and calculating from said epipolar geometry a projectionof the site from the reference image onto the target image. The locationin the target image of the site on the surface of the physical objectcan then be determined (“re-localized”) from the calculated epipolarprojections for the two or more reference images.

This approach has been found to provide an accurate and computationallyefficient for locating the site on the surface of the physical object inthe target image. The method has primarily been developed for use inendoscopy, where the reference images and the target image can be takenfrom different endoscope positions. However, the method can potentiallybe applied to a wide range of other situations, including medicalinvestigations, engineering inspections, remote sensing, etc.

In one embodiment, the method further comprises identifying a location,p_(T1), of the site in a first reference image, I₁ and a location,p_(T2), of the site in a second reference image, I₂. The epipolargeometry between the reference image I₁ and the target image T isdescribed algebraically using the fundamental matrix F_(1T):

$F_{1T} = {\begin{pmatrix}f_{1} & f_{2} & f_{3} \\f_{4} & f_{5} & f_{6} \\f_{7} & f_{8} & f_{9}\end{pmatrix} = {{{K^{- T}\left\lbrack t_{1T} \right\rbrack} - {{{}_{}^{}{}_{1T}^{}}K^{- 1}}} = {{K^{- T}\begin{bmatrix}0 & {- t_{3}} & t_{2} \\t_{3} & 0 & {- t_{1}} \\{- t_{2}} & t_{1} & 0\end{bmatrix}}R_{1T}K^{- 1}}}}$

where K is the imaging device intrinsic matrix defined with the focallength, the centre position of an image, and the scaling from 3D-spaceto the image. The epipolar geometry between the reference image I₂ andthe target image T is described algebraically using the fundamentalmatrix F_(2T):

$F_{2T} = {\begin{pmatrix}f_{1} & f_{2} & f_{3} \\f_{4} & f_{5} & f_{6} \\f_{7} & f_{8} & f_{9}\end{pmatrix} = {{{K^{- T}\left\lbrack t_{2T} \right\rbrack}_{x}\; R_{2T}K^{- 1}} = {{K^{- T}\begin{bmatrix}0 & {- t_{3}} & t_{2} \\t_{3} & 0 & {- t_{1}} \\{- t_{2}} & t_{1} & 0\end{bmatrix}}R_{2T}K^{- 1}}}}$

where K is the imaging device intrinsic matrix defined with the focallength, the centre position of an image, and the scaling from the3D-space to the image.

The epipolar projection for a reference image I₁ is calculated bycomputing the epipole e^(1T), wherein e^(1T) is the intersection of theaxes formed with the camera centre for reference image I₁ and the cameracentre for the target image with the image plane T. The epipolarprojection for a reference image 1₂ is calculated by computing theepipole e^(2T), wherein e^(2T) is the intersection of the axes formedwith the camera centre for reference image I₂ and the camera centre forthe target image with the image plane T. F_(1T)p_(T1) defines anepipolar line el₁, which passes through the projection of p_(T1) onto Tand through e^(1T), and F_(2T)p_(T2) defines an epipolar line el₂, whichpasses through the projection of p_(T2) onto T and through e^(2T). Theintersection of el₁ and el₂ corresponds to the location of the site inthe target image T.

In one embodiment, calculating the epipolar projection for a referenceimage comprises choosing multiple different subsets of the featuremappings, determining the epipolar projection for each subset, computingthe error associated with the determined epipolar projection across thewhole set of feature mappings, and selecting the determined epipolarprojection that provides the lowest computed error. Note that the totalnumber of feature mappings may be large (a hundred or more), while eachsubset might only contain several feature mappings for computationalefficiency. Such an approach may further comprise refining thedetermined epipolar projection using all of the set of feature mappingsfor the reference image.

In one embodiment, for each reference image the epipolar projectionproduces an epipolar line representing the projection in the targetimage of the location of the site in the reference image. If there aretwo reference images, the determined location for the site in the targetimage corresponds to the intersection of the two epipolar lines producedfor the two reference images. If there are three or more referenceimages, the determined location for the site in the target image can bebased on minimising a measure of distance from the three or moreepipolar lines produced for the three reference images.

In one embodiment, the uncertainty of the determined location of thesite is estimated by propagating uncertainties from the determination ofthe epipolar geometry for each reference image. Note that if three ormore reference images are used, the scatter of intersections between thethree or more epipolar lines will also give an indication of theuncertainty of the determined location.

One potential cause of error in the re-localization is any deformationbetween the various images. Nevetheless, the approach described hereinhas been found to be generally robust enough to accommodate a certainamount of deformation (which is to be expected in any endoscopicinvestigation).

In one embodiment, the method further comprises determining said set offeature mappings for each reference image in relation to the targetimage. Although in principle the feature mappings might be determined byhand (such as by a clinician marking corresponding points on twoimages), this is generally too slow for use in a real-time clinicalenvironment. Accordingly, the feature mapping is generally performedautomatically as a precursor to the re-localization.

In one embodiment, the method further comprises selecting an image touse as said target image. Thus the target imaging device may acquiremultiple images, some (or potentially all) of which do not include thedesired site on the physical object. One way of selecting a target imageis to use information about the 3-dimensional position of the targetimaging device relative to said physical object when each image isacquired. An image can then be selected as the target image if it can bedetermined from the 3-dimensional position that the target imagingdevice was viewing the desired site when the image was acquired.

In an endoscopic procedure, the information about the 3-dimensionalposition of the target imaging device relative to said physical objectmay be obtained using an electromagnetic tracking device. Such3-dimensional position information is generally accurate enough toselect an appropriate target image, but not to locate the desired sitewithin the target image with sufficient accuracy for clinical purposes.

As discussed above, the reference and target images may be acquiredduring one or more endoscopic procedures. For example, the referenceimages may have been acquired during a first endoscopic investigation,while the target image is acquired during a second endoscopicinvestigation. Alternatively, the target image may be acquired during alater phase of the same endoscopic investigation used to acquire thereference images. In either case, the approach described hereingenerally allows the location of the site in the target image to bedetermined in real-time as the target image is received from anendoscope.

A further possibility is that the target image was acquired in aninvestigation prior to the investigation used to acquire the first andsecond reference images. This might be of interest to study the historyand growth of an item (such as a cancerous growth in a medicalinvestigation, or a crack or other structural defect in an engineeringinvestigation) that is detected in the reference images, but was notpreviously noticed (and may not yet have existed) in the target image.

Various embodiments of the invention provide an apparatus and a computerprogram for implementing the above method. The computer programcomprises program instructions that when executed by a computer systemcause the computer system to perform the method. The computer programmay be stored in a computer readable storage medium, such as an opticaldisk. The apparatus may comprise a general processor for implementingsuch a program, or may comprise (in whole or in part) special-purposehardware for implementing the method.

The approach described herein provides re-localisation using robustfeature tracking and the constraints of epipolar geometry. Visualfeedback of the relocalisation can be provided in real-time on a displayof the target image. This can include an indication of a confidenceregion (e.g. 95%) around the re-localised site, which may berepresented, for example, as an ellipse shown around the site ofinterest—such as a re-localised biopsy site. This then provides theendoscopist with a region of confidence for where the true position ofthe biopsy site is.

The approach described herein can be used in any re-localisationapplication in both medical and non-medical fields. For example, in themedical field, the method can be used in endoscopic procedures andparticularly in gastroenterology, such as in oesophageal or colonicendoscopy. The method can also be used for accurate therapy delivery. Innon-medical fields, the method can be used, for example, in suchapplications as industrial inspection; remote sensing in hostileenvironments or very remote sites; remote manipulation and repair; andtelerobotics.

One embodiment of the present invention provides a method forre-localising a site comprising: determining an approximate initial3-dimensional position of the site; and determining a refined locationof the site by geometric constraint using the epipolar geometryproperties between the site in a series of at least three perspectiveprojection images. The method may further comprise determining the 95%confidence region around the re-localised site for visual feedback. Themethod may be applied to optical images, X-ray images, or any other formof image. The step of determining an initial 3-dimensional position ofthe site may be carried out using an electromagnetic device.Alternatively, a number of other technologies may be used, such asmechanical, ultrasonic or sonic positioning, optical localising,interventional MR, or manual positioning, in order to provide anapproximate position.

The approach described herein is based on the computation in the targetimage of a point's location as the intersection of two epipolar linesderived from the location of corresponding sites two or more previouslyacquired images. The epipolar geometry is recovered with a robusttechnique. Such an approach has several advantages. It provides a robustand reliable solution to track sites in any procedure. An initializationprocess can be used to give a starting point based on a tracking device,and a refinement process can then be used to obtain an accuratere-localisation position based on the geometric constraints betweenmultiple images. The method also reduces the number of computations forthe re-localization, since it only involves consideration of epipolarlines, without requiring the determination of spatial transformationsbetween the images. Therefore, the method can track/re-localise a sitewith no or minimal interaction. In contrast, for many existingendoscopic re-localisation methods, the physicians need to make a visualinspection to identify the positions and sometimes have to utilisemarkers to assist the procedure. Furthermore, the method should workefficiently in real-time, such as for use in a real-time guidancesystem, based on a information coming from the endoscopic images and theelectromagnetic tracking device. If no pre-operative images are used, nocamera tracking and/or registration is involved, but rather, sites maybe tracked directly in the endoscopic video images.

One embodiment of present invention provides a method and system forre-localisation of sites. The system for biopsy site re-localisation isbased on computer vision technology and a 3-dimensional tracking device.It includes: (1) 3D positioning for initialization: called theinitialization process. In an endoscopy application, the sensors may beattached to the tip of an endoscope (POSITION 1) and the patient's body(POSITION 2). The relative position of POSITION 1 and 2 provides aninitial location of the site and is invariable to the patient'smovement. (2) Geometric constraint to reach an accurate and reliableresult: called the refinement process. The refinement process makes useof the epipolar geometry properties between at least three images of thesame site observed from different points of view with a camera. Therefinement process includes computing two epipolar lines between the twofirst images and the third image (target image) in which the site needsto be re-localised. The intersection of these lines corresponds to theposition of the site.

Note that the use of epipolar geometry is already known in a medicalcontext, see for example Hu M. et al., “3D reconstruction of internalorgan surfaces for minimal invasive surgery”, 2007, Medical ImageComputing and Computer Assisted Intervention (MICCAI'07), Vol. 4791 ofLecture Notes in Computer Science, pp. 68-77, Springer-Verlag, but fordifferent applications (not for re-localization). The present approachfocuses on biopsy site re-localisation (and other analogous problems),which is concerned with spatial transformations (rotations andtranslations) between successive endoscopic images. As described herein,these spatial transformations can be efficiently accommodated with therecovery of the epipolar geometry formed by different endoscopic images.

BRIEF DESCRIPTION OF THE DRAWINGS

By way of example, an embodiment of the present invention will now bedescribed with reference to the accompanying drawings, in which:

FIG. 1 illustrates a method for re-localisation using epipolarprojection according to one embodiment of the present invention.

FIG. 2 illustrates how the method of FIG. 1 is extended to are-localisation with a set of N reference images (where N>2).

FIG. 3 depicts a block diagram of the site re-localisation methodaccording to one embodiment of the invention.

FIG. 4 shows an example of a re-localisation with 2 epipolar lines andincludes a 95% confidence region around the re-localised biopsy site.The right-hand image is a zoomed version of the left-hand image.

FIG. 5 shows results for four groups, each of two reference images and atarget image, of the epipoles computation, of the re-localisation, andof the errors for the fundamental matrix estimation.

FIGS. 6( a) and 6(b) illustrate examples of incorrect re-localisationdue to a smooth tissue texture, and FIGS. 6( c) and 6(d) illustrate theeffect of outliers on the epipolar line computation.

DETAILED DESCRIPTION

The following is a detailed description of an embodiment of the presentinvention as utilised in endoscopy. In particular, it is assumed that asite of interest has been acquired in two or more reference images, andthe site now has to be identified (re-localized) in a target image. Notethat the reference images and the target image may be acquired duringthe same clinical investigation (and therefore generally by the sameimaging device associated with the endoscope). Alternatively, thereference images may have been acquired in a first clinicalinvestigation and the target image is now acquired in a second clinicalinvestigation at some later date. In this latter case, the imagingdevice used to acquire the target image may be the same as or differentfrom the imaging device used to acquire the reference images. Asdescribed in more detail below, a positional sensor may be fitted at thetip of the endoscope in order to help re-localise biopsy sites when theendoscope camera moves widely.

Re-Localisation Framework

The re-localisation method is integrated into a framework comprising (a)initialization using 3D positioning information from an EM device, and(b) refinement based on geometric constraint.

a) Initialization Process

An electromagnetic (EM) position tracking device is used in associationwith the endoscope to acquire an initial, approximate location of thebiopsy site. The tracking device can be any EM tracking equipmentsuitable for a medical application, e.g., an Aurora system (NorthernDigital Inc; Waterloo, Ontario, Canada), a medSAFE system (AscensionTechnology Corp, Burlington, Vt., U.S.A.), or a ScopeGuide system(Olympus Corp, Tokyo, Japan).

During the clinical investigation, one EM sensor is attached to the tipof the endoscope so that the 3D position of the endoscope can be trackedroughly from the EM device (POSITION 1). One or more additional EMsensors are attached to the patient's body in order to provide the 3Dposition of the patient in the same EM device coordinate system(POSITION 2). The relative position of POSITION 1 and 2 can be used todetermine an approximate position of the endoscope relative to thebiopsy site (with a typical accuracy of approximately 10 mm) and isinvariate to the patient's movement

b) Refinement Process

The refinement process involves the computation of the epipolar lineswhich pass through the projection of the biopsy site in two referenceimages onto the target image.

If a biopsy site location is known in a first endoscopic image I₁(referred to as a reference image), it can be projected onto anotherendoscopic image T (referred to as a target image) for itsre-localisation in T. This projection can be determined with theepipolar geometry formed with the two endoscopic images I₁ and T. Theepipolar geometry between I₁ and T can be described algebraically usingthe fundamental matrix F_(1T) as:

$\begin{matrix}{F_{1T} = {\begin{pmatrix}f_{1} & f_{2} & f_{3} \\f_{4} & f_{5} & f_{6} \\f_{7} & f_{8} & f_{9}\end{pmatrix} = {{{K^{- T}\left\lbrack t_{1T} \right\rbrack}_{x}R_{1T}K^{- 1}} = {{K^{- T}\begin{bmatrix}0 & {- t_{3}} & t_{2} \\t_{3} & 0 & {- t_{1}} \\{- t_{2}} & t_{1} & 0\end{bmatrix}}R_{1T}K^{- 1}}}}} & (1)\end{matrix}$

K is the camera intrinsic matrix defined with the focal length, thecentre position of an image, and the scaling from the 3D-space to thecamera image. Once F_(1T) and K are known, it is possible to determinethe camera motion: the rotation R_(1T) and the translation t_(1T), withfurther computations.

During endoscopic procedures, a biopsy site can be seen from variouspoints of view with the endoscopic camera. The different viewpointsreflect different positioning of the endoscope along (and within) therelevant organ, as well as any twisting of the head of the endoscopearound its central axis. As illustrated in FIG. 1, let I₁ and I₂ be two(reference) images where the biopsy site location is visible and T be athird (target) image for which the biopsy needs to be re-localised. LetP be the biopsy site location in the 3D space, and p_(T1) and p_(T2) berespectively the locations of the biopsy site in images I₁ and I₂. Thefundamental matrices F_(1T) and F_(2T) are computed between respectivelyimages I₁ and T and images I₂ and T. The axes formed respectively withcamera centre 1 and camera centre T, and camera centre 2 and cameracentre T, have an intersection with the image plane T, which is calledthe epipole. Let e^(1T) and e^(2T) be the two epipoles of thisconfiguration. F_(1T)p_(T1) is a vector and defines the epipolar lineel₁, which passes through the projection of p_(T1) onto T and throughe^(1T). The epipolar line el₂ can be defined similarly fromF_(2T)p_(T2). As p_(T1) and p_(T2) correspond to the same biopsy sitelocation in the 3D-space, the intersection of el₁ and el₂ represents thelocation of the biopsy site in image T. This re-localisation method hasthe advantage of requiring only the computation of the fundamentalmatrices F_(1T) and F_(2T).

This re-localisation method can also be extended to a configuration of Nimages, returning a series of epipolar lines with (in theory) a commonintersection. In practice, for a configuration of N images, the epipolarlines tend not to have a unique intersection. In these circumstances,the re-localised biopsy site p can be computed by minimisation of itsperpendicular distance to the N epipolar lines, as illustrated in FIG.2, according to the following formula:

$\begin{matrix}{\min\limits_{p}{\sum\limits_{i = 1}^{N}\left( \frac{{{el}_{ix} \cdot x} + {{el}_{iy} \cdot y} + {el}_{im}}{\sqrt{{el}_{ix}^{2} + {el}_{iy}^{2}}} \right)^{2}}} & (2)\end{matrix}$

where el_(ix), el_(iy), el_(im) are the three coefficients that definethe epipolar line el_(i) with i =[1 . . . N] and x, y are thecoordinates of the re-localised biopsy site p in the target image T.

The main steps of the refinement procedure follow the computationsdescribed by R. Hartley et al. in “Multiple View Geometry in ComputerVision”, 2004, Cambridge University Press, to recover the epipolargeometry. This procedure is illustrated in FIG. 3, which depicts theprocessing where there are two reference images (1 and 2). Thisprocessing comprises two sequences: one sequence starts at Image 1 andfinishes at Image T. The other sequence starts at Image 2 and finishesat Image T. The two sequences are processed independently in steps 1, 2,3, and 4. If there are N reference images (N>2)additional sequences areused which start at Image i and finish at Image T.

The main computations for each input shown in FIG. 3 will now bedescribed in more detail.

Feature Tracking

Feature tracking is used to identify a correspondence between features(such as edges or intersections) in a reference image and matchingfeatures in the target image. For the endoscopic applications describedherein, the number of matching features between a pair of images may belarge (more than a hundred).

In one embodiment, a block matching technique is used in step 1 of FIG.3 to track reliably blocks of M×N pixels through a series of differentendoscopic images. This technique is described by K. Mori et al. in“Tracking of a bronchoscope using epipolar geometry analysis andintensity based image registration of real and virtual endoscopicimages”, Medical Image Analysis, 2002, Vol. 6, pp. 321-336, for anapplication in bronchoscopy. The similarity between two blocks in twodifferent images is measured as the cross-correlation of the pixelintensities in each block. Alternatively, a feature tracking methodbased on optical flow can also be used, such as described by B. Lucas etal. in “An iterative image registration technique with an application tostereo vision,” in Proc. IJCAI, pp. 674-679, 1981. A further possibilityis to perform the feature matching by hand, i.e. by a visual comparisonof the reference and target images.

Detection of Inliers

The Maximum A Posteriori SAmple Consensus (MAPSAC) method provides arobust method for detecting inliers (see Torr P. H. S. et al., “IMPSAC:A synthesis of importance sampling and random sample consensus”, in IEEETrans Pattern Analysis and Machine Intelligence, 25(3), pages 354-365,2003). The approach involves minimising the error in the correspondencebetween points, which is equivalent to minimising a cost function C:

$\begin{matrix}{C = {{\sum\limits_{i = {1\mspace{14mu} \ldots \mspace{14mu} n}}{{\rho \left( e_{i}^{2} \right)}\mspace{14mu} {with}\mspace{14mu} {\rho \left( e_{i}^{2} \right)}}} = \begin{Bmatrix}e_{i}^{2} & {{{if}\mspace{14mu} e_{i}^{2}} < T} \\T & {{{if}\mspace{14mu} e_{i}^{2}} \geq T}\end{Bmatrix}}} & (3)\end{matrix}$

T is a threshold set for the detection of inliers and e_(i) is thegeometric distance for a correspondence {p_(1i), p_(i)}. This techniquehas the advantage of taking into account the contribution of the inliersto the error and to the fundamental matrix computation.

Determination of the Epipolar Lines Intersection

Steps 2 and 3 from FIG. 3 are iteratively run over samples of 7correspondences S={p_(1i), p_(i)}. For each sample, the seven-pointalgorithm is applied (R. Hartley et al.) and returns one or threesolutions for the fundamental matrix F. Then the MAPSAC's cost functionis applied with the Sampson distance e_(i):

$e_{i}^{2} = \frac{\left( {p_{i\;}^{T}{Fp}_{1i}} \right)^{2}}{\left( {Fp}_{1i} \right)_{1}^{2} + \left( {Fp}_{1i} \right)_{2}^{2} + \left( {F^{T}p_{i}} \right)_{1}^{2} + \left( {F^{T}p_{i}} \right)_{2}^{2}}$

(4)where (Fp_(1i))₁ is the 1st component of the vector Fp_(1i). Thiserror expresses how well F fits the correspondences {p_(1i), p_(i)}. Atthe end of the iteration loop, F minimizes the cost C. A secondestimation of F, minimising the Sampson distance, is found in step 4 ofFIG. 3, using a constrained non-linear optimisation applied to theinliers detected from the MAPSAC. Then, in step 5 of FIG. 3, theepipolar lines passing through the projection of the biopsy site ontothe third image are determined and their intersection returns the biopsysite. If N epipolar lines are used, the biopsy site is re-localised soit minimises its perpendicular distances to the epipolar lines (as perequation 2).

Feedback Process

A 95% confidence region can be determined around the re-localised biopsysite. This is illustrated in FIG. 4, which shows two (diagonal)projected epipolar lines, and an oval representing a confidence regionaround their intersection. (N.B. the right-hand image in FIG. 4 is anenlargement of the left-hand image).

The confidence region (for 95% or any other desired level of confidence)is determined from the covariance matrix of the re-localised biopsy sitep:

$\begin{matrix}{\Lambda_{p} = {{E\left\lbrack {\left( {p - {E\lbrack p\rbrack}} \right)\left( {p - {E\lbrack p\rbrack}} \right)^{T}} \right\rbrack} = \begin{bmatrix}{var}_{px} & {cov}_{pxpy} \\{cov}_{pxpy} & {var}_{py}\end{bmatrix}}} & (5)\end{matrix}$

This covariance matrix describes the uncertainty of the re-localisedbiopsy site and can be computed by propagation of the uncertainty of thelocation of the inliers in the images firstly to the fundamental matrix,secondly to the epipolar lines, and finally to the re-localised biopsysite (see Zhang Z., “Determining the Epipolar Geometry and itsUncertainty: A Review”, in International Journal of Computer Vision,27(2), pages 161-195, 1998),

Validation

The approach described herein has been validated by first checking therecovery of the epipolar geometry and secondly assessing there-localisation error. The validation was performed directly with invivo data. These data were acquired with monocular endoscopes during agastroscopy with three patients. The image dimensions were on averageapproximately 300×300 pixels. During the procedure, the endoscopistintroduced a miniprobe into the working channel of the endoscope inorder to detect a biopsy site, touched the oesophageal tissue, andremoved it without widely moving the endoscope camera. The method hasbeen validated with eleven groups of three endoscopic images.

As the re-localisation is based on the recovery of the epipolargeometry, the computed epipole positions were checked visually everytime the method was applied to a group of three or more endoscopicimages. The endoscope motions are usually small rotations andtranslations around and along the optical axis of the camera inconsecutive images. Therefore, the epipole was expected to lie in anarea reasonably coherent with the endoscope motions. The Sampsondistance was computed for the fundamental matrix obtained firstly usinga least squares regression over all the correspondences, and secondlywith the method described herein, to check the contribution of theinliers' detection. This error was determined according to Equation (6):

$\begin{matrix}{e = \sqrt{\frac{1}{n}{\sum\limits_{i = 1}^{n}\frac{\left( {p_{i}^{T}{Fp}_{1i}} \right)^{2}}{\left( {Fp}_{1i} \right)_{1}^{2} + \left( {Fp}_{1i} \right)_{2}^{2} + \left( {F^{T}p_{i}} \right)_{1}^{2} + \left( {F^{T}p_{i}} \right)_{2}^{2}}}}} & (6)\end{matrix}$

The re-localisation method was then evaluated by the endoscopist. Apoint corresponding to a tissue region or to the tip of the miniprobewas manually selected in the first endoscopic images, tracked visually,and re-localised visually in the last image. This tracking process gavea ground-truth position, which could then be compared with the positionobtained with the re-localisation method in order to estimate the errorboth in pixels and in millimetres. As the diameter of the miniprobe was2mm, the size of a salient segment, e.g. a vessel, in the same plane asthe miniprobe could be computed. This salient segment defined a scale inthe target Image T for the conversion of the error from pixels tomillimetres and for an estimation of the tissue dimensions in the fieldof view (FOV) of the image.

The MAPSAC method was used for the detection of outliers in order tooptimise the computation of the fundamental matrix. Four results out ofeleven of epipole computation and error estimation are presented for twoanalysed regions of the first patient (Groups of images 1 and 2) and forone region of the second patient (Groups of images 3 and 4) in FIG. 5.The two epipoles from reference Image 1 and reference Image 2respectively to target Image T are described with the ‘+’ sign and theMiniprobe tip (‘×’ sign) is tracked from Image 1 and Image 2 to Image T.Two errors are computed for each group of images: one for the coupleImage 1—Image T and one for Image 2—Image T. The outlier removal gave anorder of magnitude reduction in the Sampson distance in comparison withthe least squares regression, so there is a good performance of theerror minimisation. The computed epipoles are located in an areacoherent with the endoscope motions. In the group of images 1 and 4, theepipoles computed between Image 1 and Image T, and between Image 2 andImage T, are well separated since the endoscope camera movement hasinvolved rotation and translation in all directions from Image 1 toImage 2. For the groups of images 2 and 3, where the endoscope camerahas principally moved along its optical axis, the epipoles are lessseparated. In this situation, the epipoles may coincide and the epipolarlines passing through the projection of the tracked point in Image T mayoverlay one another. Accordingly, the most accurate results aregenerally obtained when camera motions between the endoscopic images arewide enough and are not pure translations along the optical axis.

For the re-localisation validation, the miniprobe tip or a point ofinterest was manually set in Image 1 and Image 2 as indicated in FIG. 5.The estimated errors in pixels and in millimetres for the eleven groupsof three or more endoscopic images are given in Table 1 below. Thiserror varied from 2 pixels to 50 pixels in the x and y directions of theimage. This location has to be related to the dimensions in millimetresin order to take into account the three dimensions of the originalspace. Thus, the endoscopist assessed the error at a maximum of 1.5 mm.In practice, an extracted tissue sample typically has an extent of 5 mm,which means that such an error is acceptable. Indeed, when coming backwith forceps, the endoscopist would have a high chance of extracting apart of the region that had been analysed with the miniprobe.

The larger errors from 1.5 mm to 2 mm correspond mainly to endoscopicimages with a poor contrast. For example, in FIG. 6( a), the tissuetexture is smooth, so that the main feature points extracted with theblock matching are located in the tissue ridges. These points returnedless reliable correspondences than points located on vasculatures sincethe contrast varies in the ridges with the tissue motions. Thus thefundamental matrix was inaccurately estimated, which resulted in a pointautomatically re-localised at about 2 mm from the visually determinedposition (see FIG. 6( b)). Failures in the removal of outliers may alsohappen for data with good feature points, such as in the pair of imagesshown in FIGS. 6( c) and 6(d). One epipolar line computed from theminiprobe tip position in one image passed correctly through the truelocation of the tip, while another was around 1 mm from this location.Such a case resulted in an intersection of the epipolar lines at adistance of 1.5 mm from the true location.

TABLE 1 Re-localisation errors in pixels and estimation in millimeters.Dimension FOV Error Error Image (pixels) (mm) (pixels) (mm) 1 229 × 34420 × 15 1.3 × 7.6 0.60 2 229 × 344 30 × 30 12.8 × 24.7 1.39 3 229 × 34430 × 30 1.9 × 0.5 0.52 4 229 × 344 30 × 30 14.9 × 26   0.59 5 229 × 34415 × 10 47.9 × 12.4 1.28 6 193 × 235 15 × 20 29.2 × 3.4  1.40 7 193 ×235 30 × 30 12.9 × 3.6  1.22 8 193 × 235 30 × 30  2.2 × 32.5 2.56 9 193× 235 20 × 30  8.3 × 31.2 2.13 10 216 × 339 30 × 30  1.9 × 0.05 0.3 11216 × 339 30 × 30 2.1 × 0.6 0.2 12 216 × 339 20 × 20 1 × 1 0.13 13 280 ×376 20 × 70 2 × 2 0.16

Therefore, a system for the re-localisation of biopsy sites has beendisclosed. The approach described herein represents an application ofepipolar geometry properties involving a determination of thefundamental matrix. However, camera rotations and translations betweenthe various image are not computed. The validation on clinical datadescribed above shows that the re-localisation can be determined with anerror less than 1 mm.

Although the particular embodiment described above relates togastroscopic images, the method may potentially be applied to any otherendoscopic procedure or similar technique. The approach described hereinis also of potential application in a wide range of otherinvestigations, for medical, engineering, and scientific purposes, suchas remote sensing the integrity and condition of pipes and otherstructures.

It will be appreciated that the embodiment described is by way ofexample only, and that alterations or modifications may be made withinthe scope of the invention as defined in the following claims.

1-24. (canceled)
 25. A computer-implemented method for determining alocation in a target image (T) of a site on a surface of a physicalobject, the method comprising: providing two or more reference images(I₁, I₂) of said physical object that have been obtained with areference imaging device, wherein each of said two or more referenceimages includes said site on the surface of the physical object and wasobtained with the reference imaging device having a different positionand/or orientation relative to said physical object; receiving saidtarget image obtained by a target imaging device, said target imageincluding the site on the surface of the physical object; for eachreference image: using a set of feature mappings from the referenceimage to the target image to determine the epipolar geometry between thereference image and the target image, and calculating from said epipolargeometry a projection of the site from the reference image onto thetarget image; and determining from the calculated epipolar projectionsfor the two or more reference images the location in the target image ofthe site on the surface of the physical object.
 26. The method of claim25, further comprising: identifying a location, p_(T1), of the site in afirst reference image, I₁; and identifying a location, p_(T2), of thesite in a second reference image, I₂.
 27. The method of claim 26 whereinthe epipolar geometry between the reference image I₁ and the targetimage T is described algebraically using the fundamental matrix F_(1T)as: $F_{1T} = {\begin{pmatrix}f_{1} & f_{1} & f_{3} \\f_{4} & f_{5} & f_{6} \\f_{7} & f_{8} & f_{9}\end{pmatrix} = {{{K^{- T}\left\lbrack t_{1T} \right\rbrack}_{x}R_{1T}K^{- 1}} = {{K^{- T}\begin{bmatrix}0 & {- t_{2}} & t_{2} \\t_{3} & 0 & {- t_{1}} \\{- t_{2}} & t_{1} & 0\end{bmatrix}}R_{1T}K^{- 1}}}}$ where K is the imaging deviceintrinsic matrix defined with the focal length, the centre position ofan image, and the scaling from 3D-space to the image, and wherein theepipolar geometry between the reference image I₂ and the target image Tis described algebraically using the fundamental matrix F_(2T) as:$F_{2T} = {\begin{pmatrix}f_{1} & f_{2} & f_{3} \\f_{4} & f_{5} & f_{6} \\f_{7} & f_{8} & f_{9}\end{pmatrix} = {{{K^{- T}\left\lbrack t_{2T} \right\rbrack}_{x}R_{2T}K^{- 1}} = {{K^{- T}\begin{bmatrix}0 & {- t_{3}} & t_{2} \\t_{3} & 0 & {- t_{1}} \\{- t_{2}} & t_{1} & 0\end{bmatrix}}R_{2T}K^{- 1}}}}$ where K is the imaging deviceintrinsic matrix defined with the focal length, the centre position ofan image, and the scaling from the 3D-space to the image.
 28. The methodof claim 26, wherein the epipolar projection for a reference image I₁ iscalculated by computing the epipole e^(1T), wherein e^(1T) is theintersection of the axes formed with the camera centre for referenceimage I₁ and the camera centre for the target image with the image planeT, and wherein the epipolar projection for a reference image I₂ iscalculated by computing the epipole e^(2T), wherein e^(2T) is theintersection of the axes formed with the camera centre for referenceimage I₂ and the camera centre for the target image with the image planeT.
 29. The method of claim 28 wherein F_(1T)p_(T1) defines an epipolarline el₁, which passes through the projection of p_(T1) onto T andthrough e^(1T), and wherein F_(2T)p_(T2) defines an epipolar line el₂,which passes through the projection of p_(T2) onto T and through e^(2T).30. The method of claim 29 wherein the intersection of el₁ and el₂corresponds to the location of the site in the target image T.
 31. Themethod of claim 25, wherein calculating the epipolar projection for areference image comprises choosing multiple different subsets of thefeature mappings, determining the epipolar projection for each subset,computing the error associated with the determined epipolar projectionacross the whole set of feature mappings, and selecting the determinedepipolar projection that provides the lowest computed error.
 32. Themethod of claim 31, wherein calculating the epipolar projection furthercomprises refining the determined epipolar projection using all of theset of feature mappings for the reference image.
 33. The method of claim25, wherein for each reference image the epipolar projection produces anepipolar line representing the projection in the target image of thelocation of the site in the reference image.
 34. The method of claim 33,wherein there are two reference images, and the determined location forthe site in the target image corresponds to the intersection of the twoepipolar lines produced for the two reference images.
 35. The method ofclaim 33, wherein there are three or more reference images, and thedetermined location for the site in the target image is based onminimising a measure of distance from the three or more epipolar linesproduced for the three reference images.
 36. The method of claim 25,wherein the uncertainty of the determined location of the site isestimated by propagating uncertainties from the determination of theepipolar geometry for each reference image.
 37. The method of claim 25,further comprising determining said set of feature mappings for eachreference image.
 38. The method of claim 25, further comprisingselecting an image to use as said target image using information aboutthe 3-dimensional position of the target imaging device relative to saidphysical object.
 39. The method of claim 38, wherein said informationabout the 3-dimensional position of the target imaging device relativeto said physical object is obtained using an electromagnetic trackingdevice.
 40. The method of claim 25, wherein said reference imagingdevice and said target imaging device are the same device.
 41. Themethod of claim 25, wherein said reference and target images areacquired during one or more endoscopic procedures.
 42. The method ofclaim 41, wherein the location in a target image of the site isdetermined in real-time during an endoscopic procedure as the targetimage is received from an endoscope.
 43. A computer readable storagemedium containing a computer program comprising program instructionsthat when executed by a computer system cause the computer system toperform a method for determining a location in a target image (T) of asite on a surface of a physical object, the method comprising: providingtwo or more reference images (I₁, I₂) of said physical object that havebeen obtained with a reference imaging device, wherein each of said twoor more reference images includes said site on the surface of thephysical object and was obtained with the reference imaging devicehaving a different position and/or orientation relative to said physicalobject; receiving said target image obtained by a target imaging device,said target image including the site on the surface of the physicalobject; for each reference image: using a set of feature mappings fromthe reference image to the target image to determine the epipolargeometry between the reference image and the target image, andcalculating from said epipolar geometry a projection of the site fromthe reference image onto the target image; and determining from thecalculated epipolar projections for the two or more reference images thelocation in the target image of the site on the surface of the physicalobject.
 44. Apparatus for determining a location in a target image (T)of a site on a surface of a physical object, the apparatus comprising:storage means for holding two or more reference images (I₁, I₂) of saidphysical object that have been obtained with a reference imaging device,wherein each of said two or more reference images includes said site onthe surface of the physical object and was obtained with the referenceimaging device having a different position and/or orientation relativeto said physical object, and for holding said target image obtained by atarget imaging device, said target image including the site on thesurface of the physical object; and one or more processors forcomputing: for each reference image, the epipolar geometry between thereference image and the target image using a set of feature mappingsfrom the reference image to the target image, and calculating from saidepipolar geometry a projection of the site from the reference image ontothe target image; and the location in the target image of the site onthe surface of the physical object using the calculated epipolarprojections for the two or more reference images.